Encoding for remoting graphics to decoder device

ABSTRACT

A portable graphics encoder connects with one or more protocol decoder devices based on a particular communication protocol. The portable graphics encoder is not specific to any particular operating system. The portable graphics encoder receives protocol decoder device commands such as input instructions that determine higher-level graphics commands that are sent to the one or more protocol decoder devices. The higher-level graphics commands are extracted from graphics sources such as application programs. The portable graphics encoder encodes the higher-level graphics commands according to a format defined by the communication protocol, and the encoded higher-level graphics commands are sent to the one or more protocol decoder devices.

TECHNICAL FIELD

This invention relates to providing graphics commands to protocoldecoder devices such as client computers.

BACKGROUND

A server computer may host application programs and/or connect to agraphics source. Graphics commands representing graphics, from theapplication programs and/or the graphics source are remotely accessed byclient computers. A terminal service platform, such as Windows® Server2003 operating system provided by the Microsoft Corporation, is oneimplementation of such technology, where graphics commands are sent toclient computers.

The server computer is referred to as a host computer or terminalserver. The client computer is referred to as a remote terminal orremote client, and communicates with the server computer through acommunications medium such as a network. In order to communicate orexchange information (e.g., graphics commands), the server computer andclient computers may implement a communication protocol such as remotedesktop protocol (RDP) as defined by the Microsoft Corporation.

In many operating systems, such as the Windows® Server 2003 operatingsystem, application programs send relatively higher-level graphicscommands or primitives to operating system components of the servercomputer. Such higher-level graphics commands might specify or definecolors, lines, shapes, and other graphics constructs. The operatingsystem components interpret or convert such higher-level graphicscommands into relatively lower-level graphics commands or informationsuch as individual pixel values or bitmaps. Such a process of convertingfrom higher-level graphics commands to relatively lower-level graphicscommands will be referred to herein as rendering.

Application programs utilize operating system components of the servercomputer in rendering the relatively lower-level graphics commands fromhigher-level graphics commands. The operating system components areconfigured to provide the rendered relatively lower-level graphicscommands to a remote client, which utilizes this information to controlits display device.

Application programs and graphics sources are typically designed tooperate in conjunction with a local display device or monitor of theserver computer. Higher-level graphics commands from applicationprograms or graphics sources are passed to a display driver component ofthe operating system. The display driver controls a local displayadapter device/card which generates graphics on the local displaydevice. When rendering the higher-level graphics commands to relativelylower-level graphics commands, operating system components may useanother display driver that “mirrors” the drawing operations of thelocal display adapter device/card. Such a driver is called a mirrordriver. In effect, the mirror driver acts as a display driver except itdoes not generate graphics to a display. Instead, it transmits thehigher-level graphics commands to the client-computer, which then doesthe actual generating of graphics. The mirror driver, or other remotedisplay driver, may format the lower-level graphics commands intocommunication protocol (e.g., RDP) specific units that are sent to theclient computers.

The operating system components, including the display driver and mirrordriver (i.e., remote display driver), are typically integrated as partof a “kernel” or central module of the operating system. The kernel isparticular to the operating system of the server computer and is noteasily replaceable. Regardless, client computers or other devices thatare able to decode protocol specific units rely on the server computer,and in particular operating system specific components of the servercomputer to communicate and receive the protocol specific units. Inspecific, sending and receiving graphics commands relies on traditionalterminal service server and client implementations which depend onoperating system kernel components.

SUMMARY

A protocol encoder device sends higher-level graphics commands fromgraphics sources to a protocol decoder device. The higher-level graphicscommands are encoded into a format specific to a communication protocolused to connect the protocol encoder device with the protocol decoderdevice.

BRIEF DESCRIPTION OF THE CONTENTS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical items.

FIG. 1 is an illustration of a protocol encoder device—protocol decoderdevice system where multiple protocol decoder devices receivehigher-level graphics commands from one or more protocol encoderdevices.

FIG. 2 is a block diagram of protocol encoder device and protocoldecoder device components.

FIG. 3 is a block diagram of various graphics sources that provideprotocol data units to a graphics encoder included in a protocol encoderdevice.

FIG. 4 is a block diagram of a graphics encoder included in a protocolencoder device, that encodes and passes graphics PDUs and communicateswith a protocol decoder device.

FIG. 5 is a block diagram of a graphics encoder included in a protocolencoder device, that supports multiple protocol decoder devices whichaccess an application program through an application program interface.

FIG. 6 is a flow diagram illustrating establishing connections between agraphics encoder and one or more protocol decoder devices and providingencoded graphics PDUs to the protocol decoder devices.

FIG. 7 is a block diagram of an implementation of a protocol encoderdevice that processes higher-level graphics commands from graphicssources; and encodes and sends the higher-level graphics commands toprotocol decoder devices.

DETAILED DESCRIPTION

The following disclosure describes sending higher-level graphicscommands from protocol encoder devices to protocol decoder devices. Aprotocol encoder device processes the higher-level graphics commandsfrom various sources, such as application programs, video sources, andother computers.

Client-Server Computer System

FIG. 1 shows a protocol encoder device—protocol decoder devices system100. The system 100 includes one or more protocol encoder devices.Examples of protocol encoder devices include a server computer 105.Furthermore, a digital camera 110 may present a user interface (UI) to auser that might include menus, buttons, and picture-thumbnails. It wouldachieve this by injecting the graphics for these UI elements as agraphics-source. Other examples of protocol encoder devices includehandheld PCs (not shown) which would allow collaborative applicationswhere screen sharing is performed at the handheld PC.

Server computer 105 and digital camera 110 communicate with one or moreprotocol decoder devices. In this example, the protocol decoder devicesinclude a desktop computer 115, a laptop computer 120, and a handheld PC125. Communication between protocol encoder devices and protocol decoderdevices is performed using a communication protocol such as remotedesktop protocol (RDP) as defined by the Microsoft Corporation.

Desktop computer 115 is a general-purpose PC implemented with a Windows®brand operating system from the Microsoft Corporation, a Linux® brandoperating system, or other operating system. The desktop computer 115 isa standalone computer that primarily interfaces to server computer 105to access files or other information (e.g., application programs hostedon server computer 105) that are not locally stored.

Laptop computer 120 is configured with its own operating system,processing unit, and storage system. The laptop computer 120 maylikewise be configured to run a Windows® brand operating system, aLinux® operating system, or other operating system.

Handheld PC 125 may possess less functionality than a general-purposecomputer. Handheld PC 125 may be equipped with one of various operatingsystems including a Windows® brand operating system, such as Windows® CEoperating system from the Microsoft Corporation.

A network 130 connects the protocol encoder devices with the protocoldecoder devices. Furthermore, the network 130 connection between thedevices may implement a transport protocol such as transmission controlprotocol over Internet protocol (TCP/IP). In other implementations adirect stream oriented connection may be established between protocolencoder devices and protocol decoder devices.

Protocol encoder devices such as the server computer 105 and/or digitalcamera 110 send higher-level graphics commands which are encoded per thecommunication protocol to protocol decoder devices (i.e., computers 115,120, and 125). The higher-level graphics commands allow the generatingof graphics at the protocol decoder devices. As further discussed below,protocol encoder devices such as the server computer 105 receivehigher-level graphics commands from graphics sources that may includeapplication programs, or from various graphics sources such as othercomputers, and encode the higher-level graphics commands per the definedcommunication protocol.

The computers 115, 120, and 125 may be based on different operatingsystems; however, the computers 115, 120, and 125 are able to receiveand generate graphics from the higher-level graphics commands receivedfrom the protocol encoder devices (e.g., server computer 105 and/ordigital camera 110).

The system 100 is representative of many different architecturesincluding direct dialup via modem, enterprise LANs (local areanetworks), WANs (wide area networks) and the Internet. The network 130may be implemented in a number of ways to support such networkingcontexts, including both wired-based technologies and wirelesstechnologies. Aspects of this invention are not limited to one specificnetwork architecture or network technology.

The server computer 105 may be implemented as a Windows® Server 2003,Windows® NT server, Linux®, or any other operating system. A moredetailed description of the server computer 105 is given below withrespect to FIG. 7. The communication protocol (e.g., RDP) used byprotocol encoder devices and protocol decoder devices may provide forencoded protocol data units (PDU) defined as having a particular dataformat. The encoded PDUs are particularly used to provide information toand from the protocol encoder devices and the protocol decoder devices.For example, higher-level graphics commands from the server computer 105are provided as encoded PDUs, and requests from computers 115, 120, and125 are provided as encoded PDUs.

A display device 135, which includes a display and input devices such asa keyboard and mouse, connects locally to server computer 105. Displaydevice 135 provides user input to application programs hosted by servercomputer 105 and displays generated graphics from server computer 105.

A graphics source 140 which includes remote computers (i.e., sourcesdisplaying graphics) may provide higher-level graphics commands to theserver computer 105. It is contemplated that graphics source 140 may beimplemented as hardware, software, firmware or a combination. Graphicssource 140 may be any source that generates a list of graphics orders.Graphics source 140 may be connected to server computer 105 through oneof various physical connections as discussed below. Furthermore,graphics source 140 may be connected though one or more networks toserver computer 105. In certain implementations, graphics source 140 ispart of server computer. In specific, when application programs areresident on server computer 105, graphics source 140 may be implementedas part of server computer 105.

Device Architectures

FIG. 2 shows example top level architectures at a protocol encoderdevice and a protocol decoder device. A graphics source 200 such asgraphics source 140 is connected to a protocol encoder device 205 (e.g.,server computer 105). Protocol encoder device 205 may be implemented ashardware, software, firmware or a combination. For example, it iscontemplated that for certain implementations, protocol encoder 205 is asoftware program included in a hardware device. In particularimplementations, the graphics source 200 is part of protocol encoderdevice 205, such as when application programs are resident on servercomputer 105 or when digital camera 110 includes the graphics source200. Graphics source 200 in particular provides for graphics protocoldata units (PDUs) as defined by the communication protocol. Such PDUsare not encoded and are received by the protocol encoder device 205 andencoded per the communication protocol by a graphics encoder 210.

Application programs resident on server computer 105 and graphicssources connected (e.g., graphics source 140) to server computer 105specify graphics in terms of relatively higher-level graphics commandssuch as RDP commands, and particularly formatted PDUs.

When graphics are generated locally at the protocol encoder device 205,higher-level graphics commands are processed or extracted from graphicssource 200 by a local graphics driver 215. Since graphics source 200provides formatted PDUs, the formatted PDUs are decoded prior to localgraphics generating. Local graphics driver 215 uses the higher-levelgraphics commands to generate graphics which are displayed on displaydevice 220. Furthermore, local graphics driver 215 is specific to theoperating system implemented by protocol encoder device 205 and may ormay not reside in the kernel. It is contemplated that local graphicsdriver 215 may be replaceable. Local graphics driver 215 is particularlyused in screen sharing implementations where higher-level graphicscommands are sent to protocol decoder device 215.

In certain cases, higher-level graphics commands from graphics source200 are received by graphics encoder 210. In particular protocolspecific PDUs are provided by graphics source 200 and received bygraphics encoder 210 for encoding in a protocol specific format and sentto protocol decoder device 225. Graphics encoder 225 is configured to bea portable component part of the operating system kernel of protocolencoder device 205. Graphics encoder 210 may be loaded or created by theprotocol encoder device 205 when communication is performed withprotocol decoder device 225. Graphics encoder 210 is further describedin different in implementations FIGS. 4 and 5. In general, higher-levelgraphics commands from graphics source 200 in the form of PDUs arereceived by graphics encoder 210 which encodes the higher-level graphicscommands into a format defined by the communication protocol used byprotocol encoder device 205 and protocol decoder device 225.

The encoded (i.e., formatted) higher-level graphics commands arereceived by protocol decoder device 225. Protocol decoder device 225includes a graphics component 230, which may further include a graphicsdriver component and a graphics generator component. In particular,graphics component 230 decodes the formatted higher-level graphicscommands and generates graphics from decoded higher-level graphicscommands. The generated graphics are displayed on a local display device235.

Graphics Sources

FIG. 3 shows examples of graphics source architectures. Graphics sources300 may describe graphics sources 140 and 200. In one implementation, agraphics source 300(1) includes an application program 305 from whichhigher-level graphics commands are extracted. The higher-level graphicscommands may be extracted through application program interface callscall such as “Win32 API” calls defined by Windows® Terminal Servicesplatform.

Application program 305 calls an operating system (OS) renderingcomponent 315, which may be a graphics display interface (GDI) componentas defined by Windows® Terminal Services platform. OS renderingcomponent 315 may be used to generate graphics locally.

The OS rendering component 315 in turn calls a protocol encoding mirrordisplay driver 325 which produces graphics PDUs representative of thehigher-level graphics commands. As described below, the graphics PDUsare sent to, encoded by, and sent by a graphics encoder to a protocoldecoder device.

In another implementation, a graphics source 300(2) includes a protocolencoding server display driver 330 which replaces and performs theencoding functions of the protocol encoding mirror display driver 325.In this implementation, the protocol encoding server display driver 330is a local display driver used in generating graphics at the protocolencoder device.

Another implementation is graphics source 300(3) where an injectionapplication program interface 335 receives the higher-level graphicscommands from application program 305. The injection application programinterface 335 sends the higher-level graphics commands to protocolencoding engine 340 which produces graphics PDUs representative of thehigher-level graphics commands.

Injection application program interface 335 is configured to process orextract higher-level graphics commands from application program 305 or agraphics source, where a graphics source may be any source displayinggraphics (e.g., video) from which graphics commands or higher-levelgraphics commands are extracted.

Exemplary uses of graphics injection using injection application programinterface 335 include accessing graphics from a graphics source ordisplaying information such as stock reports, news, etc. In theseimplementations, the graphics source may be separate from protocolencoder device (e.g., server computer). A graphics-source may begenerated within an application. For instance, an application may statethat it wants the protocol encoding engine 340 to encode a red square inthe middle of a black screen for display at a decoder device. This couldbe useful in systems that do not use a prior art complex graphics-systemand provide an ability to generate a simple set of drawings for display.The Graphic injection application performs provides this ability bysending a PDU to a protocol state machine. In this example, the PDU willbe of the type “TS_UPDATETYPE_ORDERS” and contains two protocol graphicorders of type “TS_ENC_OPAQUERECT_ORDER” where the first opaquerectangle order will be painted on the entire surface and the secondopaque order will paint the red square.

Furthermore, the graphics source need not be based on the operatingsystem(s) or hardware platform(s) implemented by the protocol encoderdevice or protocol decoder device as long as higher-level graphicscommands can be extracted using injection application program interface335. This allows for the ability of protocol encoder devices andprotocol decoder devices to access graphics (i.e., higher-level graphicscommands) from other devices running different operating systems.

In another exemplary implementation, a graphics source 300(4) implementsa screen scraping engine and protocol encoder 345. Screen scrapingengine and protocol encoder 345 acts as a mirror driver and encodeshigher-level graphics commands sent to protocol decoder devices. Themirroring applies to a local graphics driver of protocol encoder device.The local graphics driver generates graphics displayed locally. Thehigher-level graphics commands used in generating graphics locally aremirrored by the screen scraping engine and protocol encoder 345. Themirrored higher-level graphics commands are then encoded as graphicsPDUs by screen scraping engine and protocol encoder 345.

For implementations of the graphics source 300, the graphics PDUs thatare sent may include a string of graphics PDUs. In certain cases, thefinal PDU of the string of PDUs may be modified to indicate a “last” orfinal PDU of the string of PDUs.

Graphics Encoder

FIG. 4 shows an example graphics encoder that receives and encodesgraphics PDUs, and communicates with a protocol decoder device. FIG. 4is an exemplary implementation and it is noted that the describedcomponents may be implemented using different architectures. Forexample, the described components may be implemented as computer programroutines or as executable functions such as DLLs (dynamic librarylinks).

Graphics PDUs from graphics source 300 are received by a graphicsencoder 400. Graphics encoder 400 is included in a protocol encoderdevice (e.g., protocol encoder device 205), and as described above isconfigured to be portable between different operating systems. In otherwords, different operating systems may use graphics encoder 400.

A stream-oriented connection 405 may be used to connect graphics encoder400 with protocol decoder device 220. As described above a networkconnection may also be implemented. In this example, stream-orientedconnection 405 includes encoded graphics PDUs sent from graphics encoder400 and encoded PDUs sent from protocol decoder device 220. Thestream-oriented connection 405 includes PDUs that represent higher-levelgraphics commands from graphics encoder 400 and information such asinput commands from the protocol decoder device 220 to graphics encoder400. In certain implementations, stream-oriented connection 405 may beimplemented using various network or transmission protocols such astransmission control protocol over internet protocol (TCP/IP); namedpipe stream protocols; or shared memory protocols. PDUs sent throughstream-oriented connection 405 are received in the order that they aresent.

Graphics PDUs are received by a graphics PDU packing component 410 whichpackages the graphics PDUs into lower level protocol units 415 that arereceived by a protocol compliant state machine 420. Lower level protocolunits 415 are defined and formatted per the particular communicationprotocol (e.g., RDP) that is implemented.

Protocol compliant state machine 420 specifically communicates withprotocol decoder device 220 using the implemented communicationprotocol. Protocol compliant state machine 420 may be implemented insoftware, hardware, firmware, or a combination. Although based on aparticular communication protocol, protocol compliant state machine 420is not restricted to current and specific implementation of thecommunication protocol and may be updated to support changes in thecommunication protocol. Protocol compliant state machine 420 may beinstalled or loaded when a graphics source (e.g., graphics source 300)attempts to send higher-level graphics commands (i.e., graphics PDUs) tographics encoder 400.

Protocol compliant state machine 420 negotiates with protocol decoderdevice 420 as to compression, encryption, and graphics encodingparameters based on a specification of the communication protocol (e.g.,RDP specification). When an input command in the form of an encoded PDUis received from the protocol decoder device 420, protocol compliantstate machine 420 decodes the PDU and forwards it as input PDU 425 to anoperating system specific input translator 430.

Operating system specific input translator 430 is specific to theoperating system of the protocol encoder device, and is replaceable(i.e., if encoder 400 is ported over to another operating system,operating system specific input translator 430 may be replaced).Protocol decoder device 220 may send input information such askeystrokes or mouse movement. The input information is described usingthe particular communication protocol. Operating system specific inputtranslator 430 translates the communication protocol formatted inputinformation sent by the protocol decoder device 220 into actions or datastructures specific to the operating system of the protocol encoderdevice.

Encoding control logic 435 provides control and defined functionalityfor the components of graphics encoder 400. In particular, encodingcontrol logic 435 provides control interfaces 440, 445, 450 respectivelyto protocol compliant state machine 420, operating system specific inputtranslator 430, and graphics PDU packing component 410. An example ofcontrol is when graphics source 300 sends graphics PDUs to graphics PDUpacking component 410. The formatted graphics PDUs cannot be sent fromgraphics PDU packing component 410 to protocol compliant state machine420, until the protocol decoder device 220 is authorized. Since theprotocol compliant state machine 420 performs negotiating with theprotocol decoder device 220, communication protocol events 455 arepassed to the encoding control logic 435 indicating authorization ofprotocol decoder device 110. Other functions that may be provided byencoding control logic 435 include authenticating and managingconnections with protocol decoder device 220, and setting up and gettingparameters for such connections.

Intermediate Language

For certain implementations, an intermediate language is used to conveyhigher-level graphics commands. Communication protocols such as the RDPcommunication protocol may include complicated protocol orders such asstate synchronization commands between protocol encoder devices andprotocol decoder devices. Furthermore, if graphics source 300(3) whichincludes injection application program interface 335 is used, graphicssource 300(3) may need to have an intimate knowledge of thecommunication protocol in order to provide valid higher-level graphicscommands as defined by the communication protocol.

Therefore the intermediate language may be implemented at graphicssource 300. The intermediate language may include simplified syntax orsyntax more applicable to the graphics source 300. Such syntax may bedifferent than the syntax of the language defined by the particularcommunication protocol. In this implementation, an intermediate languageencoder may be included in graphics encoder 410. The intermediatelanguage encoder the received intermediate language graphics commandsinto graphics PDUs as defined by the communication protocol.

An example of drawing commands in an intermediate language is thefollowing which describes red square in the middle of a black screen:

var oFrame=Encoder.CreateFrame( );

Frame.DrawRec(0, 0, ScreenSize.x, ScreenSize.y, BLACK);

Frame.DrawRect(64, 64, ScreenSize.x-64, ScreenSize.y-64, RED);

Frame.Flush( );

Application Program Sharing

FIG. 5 shows a graphics encoder 215 that supports multiple clientcomputers 110 (i.e., protocol decoder devices) accessing an applicationprogram 500. The application program 500 sends higher-level graphicscommands and receives client computer 110 commands and instructionsthrough a connection 505. In this example, server computer 105 includesgraphics encoder 400 which connects with client computers 110.

An application sharing communication application program interface (API)510 connects with application program 500. In this implementation, API510 connects with graphics encoder 215 via a connection 515 which may bea programmatic interface such as a C++ language interface. In otherimplementations, API 510 is integrated as part of graphics encoder 215.

API 510 passes input commands to application program 500 from authorizedclient computers 110. API 510 sends to authorized client computers 110higher-level graphics commands from application program 500, such thatauthorized client computers 110 view the same graphics (i.e., generatethe same graphics using the same higher-level graphics commands). Thisuse is particularly of benefit in multi-party conference situations.

Graphics encoder 400 includes dedicated protocol compliant statemachines 525(1)-525(N) which respectively support client computers110(1)-110(N). Protocol compliant state machines 525 perform similarfunctions as protocol compliant state machine 310 describe above, suchas negotiating with client computers 110 as to compression, encryption,and graphics encoding as defined by a particular communication protocol.Dedicated protocol compliant state machines 525 allow for each clientcomputer 110 to have a different negotiated set of properties, forexample the property of color-depth. Thus, a client computer 110 thatonly supports low-color-depth can get a degraded experience whileanother client computer 110 better can get the full color-richnessexperience.

Furthermore, if one of the client computers 110 has a relatively slowlink between itself and the server computer 105, it is advantageous tonot have the slower client computer 110 affect (i.e., “bottleneck”)transmission to the other client computers 110. For instance, if videois being shown, and one client computer 110 can only receive one frameper-second, the other client computers 110 which are capable to receiveare a greater rate of frames are not limited by the “slower” clientcomputer 110.

Stream-oriented connections 530(1)-530(N) connect client computers 110with graphics encoder 400 by way of protocol compliant state machines525. The stream-oriented connections 530 are similar to stream-orientedconnection 305 described above, and include information in the form offormatted PDUs.

Each of the protocol compliant state machines 525 provides inputcommands from client computers 110 as formatted communication protocolevents 535 to encoding control logic 435. Encoding control logic 435provides a control interface 540 and sends graphics PDUs 545 to each ofprotocol compliant state machines 525.

Certain client computers 110 have particular access rights to (i.e.,usage) application program 500. In other words, certain client computers110 may have complete access rights to application program 500, whileother client computers 110 may have limited access rights to applicationprogram 500. Using the example of a multi-party conference situation,certain client computers 110 may be moderators and are able to viewgraphics from and provide input to application program 500, while otherclient computers 110 may be viewers that are only able to view graphics(i.e., receive higher-level graphics commands) from application program500.

Sharing parameters component 550 defines access rights of each of clientcomputers 110, such as sharing of a color depth. Shared area filteringcomponent 555 is used to send client computers 110 a portion of a sharedscreen (i.e., particular higher-level graphics commands related to aportion of a screen display). Through the API 510 the shared areafiltering component 555 provides the particular higher-level graphicscommands.

A participant management component 560 manages connection of clientcomputers 110, and can also instruct the API 510 as to which clientcomputers 110 are connected and disconnecting client computers 110.Connection events 565 are received from encoding control logic 435,which include requests sent from client computers 110 to connect withgraphics encoder 215 (i.e., server computer 105). Participant managementcomponent 560 sends connection control commands 570 to encoding controllogic 435, which are passed on to client computers 110 to allow andmaintain connection with graphics encoder 400 (i.e., server computer105).

A user mode input translator 575 receives input PDUs 580 which includesinput from client computers 110. Input translator 575 performs functionssimilar to operating system specific input translator 430. Inparticular, input translator 575 translates the input PDUs 580 sent bythe client computer 110 into actions or data structures as defined andunderstood by the operating system of server computer 105.

An application sharing screen scraping driver 585 performs similarfunctions as screen scraping engine and protocol encoder 345 describedin FIG. 3. In particular, application sharing screen scraping driver 585is used to extract or mirror higher-level graphics commands from a localgraphics driver and encodes the extracted higher-level graphics commandsas graphics PDUs 590. The graphics PDUs 590 are sent to encoding controllogic 435 which passes them on to protocol compliant state machines 525in the form of graphics PDUs 545.

FIG. 6 shows a process 600 to establish connections between a protocolencoder device and one or more protocol decoder devices, and providehigher-level graphics commands to the protocol decoder devices.

The process 600 is illustrated as a collection of blocks in a logicalflow graph, which represent a sequence of operations that can beimplemented in hardware, software, firmware, or a combination thereof.In the context of software, the blocks represent computer instructionsthat, when executed by one or more processors, perform the recitedoperations. The process 600 is described with reference to protocolencoder device such as server computer 105 of FIG. 1, and particularlyto operating system portable graphics encoder 400 of FIG. 4, whereserver computer 105 and graphics encoder 400 provides for means toperform particular processes.

Although described as a flowchart, it is contemplated that processes maytake place concurrently. For example, sending encoded graphics PDUs andthe processing of graphics PDUs may be two concurrent actions which maytake place in a loop until a protocol decoder device is disconnected orthe protocol encoder device decides to stop sending graphics PDUs and/orstop processing input.

At block 605, a protocol encoder device through a graphics encoderestablishes a connection with one or more protocol decoder devices. Agraphics encoder of the protocol encoder device may establish connectionwith the protocol decoder devices through protocol compliant statemachines dedicated to each protocol decoder device. Establishing aconnection may be initiated when an application program or graphicssource attempts to send higher-level graphics commands or graphicsprotocol data units (PDU) to protocol decoder device. In other cases, aconnection may be initiated from a protocol decoder device requesting toconnect to the protocol encoder device. The connection is performedusing a particular communication protocol such as RDP communicationprotocol, where information exchanged between the server computer (i.e.,the protocol compliant state machine) and the protocol decoder device(s)may be in the form of encoded and formatted PDUs. The exchangedinformation may be through a stream-oriented connection which may or maynot incorporate a network.

When establishing connection with one or more protocol decoder devices,the dedicated protocol compliant state machine(s) may be loaded (i.e.,installed) in the graphics encoder to establish communication with therespective protocol decoder devices. The protocol compliant statemachine(s) may be loaded when an application program or graphics sourceattempts to send higher-level graphics commands (graphics PDUs) to theprotocol decoder device(s).

At block 610, the application program or graphics source is directed orinformed to begin sending higher-level graphics commands (graphics PDUs)to the graphics encoder. An application program interface (API) may beused to provide information to and access the higher-level graphicscommands from the application program or graphics source.

For certain implementations, higher-level graphics commands from theapplication program or graphics source are used to generate and displaygraphics to a local display device. The higher-level graphics commandsare particularly used by a local graphics driver to generate graphicsdisplayed on the local display device. Higher-level graphics commandsare derived from the local graphics driver (i.e., following the “YES”branch of block 615). This implementation applies as to screen sharingsituations.

At block 620, a screen scraping engine and protocol encoder extracts ormirrors the higher-level graphics commands that are used to generategraphics at the local graphics driver. The screen scraping engine andprotocol encoder encodes the higher-level graphics commands into aformat as defined by the communication protocol. In certain cases, whenthe application program or graphics source implement an intermediatelanguage that has different syntax than the language defined by thecommunication protocol, an intermediate language encoder may beimplemented to translate intermediate language higher-level graphicscommands to the language or format as defined by communication protocol.During the establishing connection, access privileges of protocoldecoder devices may be determined. Access privileges include the abilityof protocol decoder devices to provide inputs and view graphics (i.e.,receive higher-level graphics commands related to certain graphics).

At block 625, input commands from the protocol decoder devices may bereceived. Input commands may represent an action or instruction to beperformed on an application program from which higher-level graphicscommands are received. Certain protocol decoder devices may beauthorized to provide inputs (e.g., commands) to application programs.In certain cases, no inputs are received from the protocol decoderdevices; for example, when higher-level graphics commands are sent froma graphics source such as a video display source. An operating systemspecific input translator receives input commands in the form ofcommunication based protocol information and translates such commandsinto operating system specific actions or data structures that supportthe operating system of the protocol encoder device.

At block 630, higher-level graphics commands are sent to protocoldecoder devices. The higher-level graphics commands are formatted tosupport the particular communication protocol and may be sent throughthe stream-oriented connection by protocol compliant state machine(s)supporting each of the protocol decoder devices. In specific, thehigher-level graphics commands are sent as encoded formatted PDUs. Theencoded formatted PDUs are decoded at the protocol decoder devices intohigher-level graphics commands used to generate graphics.

A graphics injection implementation may take place when higher-levelgraphics commands are not derived or received from a graphics driver(i.e., following the “NO” branch of block 615). Graphics injectioninvolves receiving higher-level graphics commands through a graphicsinjection application program interface (API).

At block 635, higher-level graphics commands are injected (i.e.,received) from the graphics injection API connected to the applicationprogram to a protocol encoding engine which provides the graphicsencoder with graphics PDUs or graphics source into a graphics injectionapplication.

Server Computer

FIG. 7 shows an example implementation of a protocol encoder device, andin particular server computer 105. Other protocol encoder devices mayimplement different architectures; however, it is contemplated that theprotocol encoder devices are able to support or port over the describedgraphics encoder 400 regardless of the operating system that isimplemented by the protocol encoder devices. As appreciated by thoseskilled in the art, certain protocol encoder devices such as digitalcamera 110 may forego or implement different hardware components thanare described in FIG. 7.

Server computer 105 may be configured with a Windows® brand operatingsystem. The server computer 105 includes a processing unit 705, a systemmemory 710, and a system bus 715 that interconnects various systemcomponents, including the system memory 710 to the processing unit 705.The system bus 715 may be implemented as any one of several busstructures and using any of a variety of bus architectures, including amemory bus or memory controller, a peripheral bus, and a local bus.

The system memory 710 includes read only memory (ROM) 720 and randomaccess memory (RAM) 725. A basic input/output system 730 (BIOS) isstored in ROM 720.

The server computer 105 has one or more of the following drives: a harddisk drive 730 for reading from and writing to a hard disk or hard diskarray, a magnetic disk drive 735 for reading from or writing to aremovable magnetic disk 740, and an optical disk drive 745 for readingfrom or writing to a removable optical disk 750 such as a CD ROM orother optical media. The hard disk drive 730, magnetic disk drive 735,and optical disk drive 745 are connected to the system bus 715 by a harddisk drive interface 760, a magnetic disk drive interface 765, and anoptical drive interface 770, respectively. The drives and theirassociated computer-readable media provide nonvolatile storage ofcomputer readable instructions, data structures, program modules andother data for server computer 105.

Although hard disk 730, removable magnetic disk 735, and removableoptical disk 750 are described, other types of computer readable mediacan be used to store data. Other such media include magnetic cassettes,flash memory cards, digital video disks, Bernoulli cartridges, randomaccess memories (RAMs), read only memories (ROMs), and the like.Additionally, the server computer 105 may be configured to serve datastored on an independent system, such as a RAID (redundant array ofindependent disks) storage system, particularly when implemented as aterminal server.

A number of program modules may be stored on the hard disk 730, magneticdisk 735, optical disk 750, ROM 720, or RAM 725. The programs include aserver operating system 775, one or more application programs 780, otherprogram modules 782, and program data 784.

A user may enter commands and information into the server computer 105through input devices such as keyboard 786 and a mouse 788. Other inputdevices (not shown) may include a microphone, joystick, game pad,satellite dish, scanner, and the like. These and other input devices,such as the graphics source 125 of FIG. 1, are connected to theprocessing unit 705 through a serial port interface 790 that is coupledto the system bus 715, but may alternatively be connected by otherinterfaces, such a parallel port, game port, or a universal serial bus(USB).

A monitor 792 or other type of display is also connected to the systembus 715 via an interface, such as a video adapter card 794. The servercomputer 105 has a network interface or adapter 796, a modem 798 orother means for establishing communications over the network 115, suchas an Internet connection. The modem 798 may also facilitate connectionfrom a protocol decoder device. Monitor 792 and input devices such askeyboard 786 and mouse 788 may be grouped and considered as a displaydevice, such as display device 135 of FIG. 1.

CONCLUSION

The above-described server computer provides higher-level graphicscommands to protocol decoder devices through a portable graphics encoderthat may be implemented by various operating systems and platforms. Thehigher-level graphics commands are encoded as to a particular formatdefined by a communication protocol used between graphics encoder andthe one or more protocol decoder devices. Although the invention hasbeen described in language specific to structural features and/ormethodological acts, it is to be understood that the invention definedin the appended claims is not necessarily limited to the specificfeatures or acts described. Rather, the specific features and acts aredisclosed; as exemplary forms of implementing the claimed invention.

1. A portable graphics encoder comprising: a protocol compliant statemachine that communicates with a protocol decoder device using acommunication protocol; a replaceable operating system specific inputtranslator that receives protocol decoder device input commands from theprotocol compliant state machine; a protocol data unit (PDU) packingcomponent that packages graphics PDUs received from a graphics sourceand encodes the packaged PDUs to the protocol compliant state machine;and control logic to manage connection between the protocol compliantstate machine, receipt of protocol decoder device commands, and sendingof encoded graphics PDUs to the protocol decoder device.
 2. The portablegraphics encoder of claim 16 wherein the protocol compliant statemachine communicates with the protocol decoder device through astream-oriented connection.
 3. The portable graphics encoder of claim 1wherein the protocol compliant state machine negotiates with theprotocol decoder device as to compression, encryption, and encodingparameters.
 4. The portable graphics encoder of claim 1 wherein theprotocol compliant state machine receives encoded PDUs from the protocoldecoder device that describe protocol decoder device inputs.
 5. Theportable graphics encoder of claim 1 wherein the operating systemspecific input translator receives PDUs which are translated to actionsspecific to an operating system used by a protocol encoder device thatincludes the portable graphics encoder.
 6. The portable graphics encoderof claim 1 wherein the PDU packing component encodes the higher-levelgraphics commands into a communication protocol defined format.
 7. Theportable graphics encoder of claim 1 wherein the control logicauthenticates the protocol decoder device.
 8. The portable graphicsencoder of claim 1 wherein the control logic manages connections withthe protocol decoder device.
 9. A method of sending higher-levelgraphics commands to one or more protocol decoder devices comprising:establishing a connection with the one or more protocol decoder devices;informing a graphics source that the connection is established;directing the graphics source to send the higher-level graphicscommands; formatting the higher-level graphics commands to allow theprotocol decoder devices to receive the higher-level graphics commands.10. The method of claim 9 wherein the establishing a connection is basedon a communication protocol.
 11. The method of claim 9 wherein theestablishing a connection is initiated when the graphics source attemptsto send a higher-level graphics command to the one or more protocoldecoder devices.
 12. The method of claim 9 wherein the establishing aconnection is initiated when the one or more of protocol decoder devicesrequests to connect.
 13. The method of claim 9 wherein the establishinga connection sets up a stream-oriented connection to the one or moreprotocol decoder devices.
 14. The method of claim 9 wherein theestablishing a connection loads dedicated protocol compliant statemachines that communicate with the one or more protocol decoder devices.15. The method of claim 9 wherein the establishing a connectiondetermines authorization access privileges of the one or more protocoldecoder devices.
 16. The method of claim 9 wherein the informing isperformed through an application program interface that connects to thegraphics source.
 17. The method of claim 9 wherein the directing isperformed through an application program interface that connects to thegraphics source.
 18. The method of claim 9 wherein the formattingcomprises encoding the higher-level graphics commands into communicationprotocol data units.
 19. The method of claim 9 further comprisingreceiving input commands from the one more protocol decoder devices. 20.The method of claim 19 further comprising translating the input commandsinto actions specific to the one or more protocol decoder devices whichthe input commands are received from.
 21. The method of claim 9 whereinthe higher-level graphics commands are derived from a graphics driverused to generate graphics locally.
 22. The method of claim 9 wherein thehigher-level graphics commands are received from a graphics injectioninterface.
 23. One or more computer-readable media comprisingcomputer-executable instructions that, when executed, perform the methodas recited in claim
 1. 24. For use with a portable graphics encoder, astorage medium having instructions that, when executed on the portablegraphics encoder, performs acts comprising: connecting with one or moreprotocol decoder devices based on a communication protocol; receivinghigher-level graphics commands from a graphics source wherein thehigher-level graphics commands are destined to the one or more protocoldecoder devices; encoding the higher-level graphics commands based on aformat defined by the communication protocol; and sending the encodedhigher-level graphics commands to the one or more protocol decoderdevices.
 25. A storage medium as recited in claim 24 wherein theconnecting is further based on a transmission control protocol.
 26. Astorage medium as recited in claim 24 wherein the receiving includesextracting higher-level graphics commands from a graphics driver used togenerate graphics locally.
 27. A storage medium as recited in claim 24further comprising receiving input commands from the one or moreprotocol decoder devices, that determine higher-level graphics commandsthat are sent to the one or more protocol decoder devices.
 28. A storagemedium as recited in claim 24 further comprising authenticating the oneor more protocol decoder devices as to access rights to provide inputsand receive higher-level graphics commands.
 29. A transmission ofhigher-level graphics commands by a portable graphics encodercomprising: connecting means to connect with one or more protocoldecoder devices; instructing means to instruct a graphics source to sendhigher-level graphics commands; encoding means to encode thehigher-level graphics commands; and sending means to send the encodedhigher-level graphics commands to the one or more protocol decoderdevices.
 30. The transmission as recited in claim 29 further comprisingreceiving means to receive commands from the one or more protocoldecoder devices to affect the higher-level graphics commands which aresent to the one or more protocol decoder devices.
 31. The transmissionas recited in claim 29 further comprising authorizing means to authorizethe one or more protocol decoder devices to receive the higher-levelgraphics commands and/or provide input commands.